Robust Multi-Agent Coordination via Evolutionary Generation of Auxiliary Adversarial Attackers

Cooperative Multi-agent Reinforcement Learning (CMARL) has shown to be promising for many real-world applications. Previous works mainly focus on improving coordination ability via solving MARL-specific challenges (e.g., non-stationarity, credit assignment, scalability), but ignore the policy perturbation issue when testing in a different environment. This hasn't been considered problem formulation or efficient algorithm design. To address this issue, we firstly model as Limited Policy Adversary Dec-POMDP (LPA-Dec-POMDP), where some coordinators from team might accidentally and unpredictably encounter limited number of malicious action attacks, regular still strive intended goal. Then, propose Robust Multi-Agent Coordination Evolutionary Generation Auxiliary Adversarial Attackers (ROMANCE), which enables trained diversified strong auxiliary adversarial attacks during training, thus achieving high robustness under various perturbations. Concretely, avoid ego-system overfitting specific attacker, maintain set attackers, is optimized guarantee attackers attacking quality behavior diversity. The goal minimize effect, novel diversity regularizer based sparse applied diversify behaviors among attackers. then paired with population selected maintained attacker set, alternately against constantly evolving Extensive experiments multiple scenarios SMAC indicate our ROMANCE provides comparable better generalization than other baselines.